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ABSTRACT 

' We study the clustering properties of about 1200 z ~ 4 Lyman Break Galaxy (LBG) candidates 

, with i' < 26 which are selected by color from deep BRi' imaging data of a 618 arcmin^ area in the 

■ Subaru/XMM- Newton Deep Field taken with Subaru Prime Focus Camera. The contamination and 
{3J)| completeness of our LBG sample are evaluated, on the basis of the Hubble Deep Field North (HDFN) 
^ . objects, to be 17% and 45%, respectively. We derive the angular correlation function over 9 = 2" — 1000", 

■ and find that it is fitted fairly well by a power law, u>{9) = A;^9~°-^, with = 0.71 ± 0.26. Wc then 
calculate the correlation length (in comoving units) of the two-point spatial correlation function 
f(r) — {r/ro)~^-^ from A^^ using the redshift distribution of LBGs derived from the HDFN, and find 
To = 2.7^'^Q g Mpc in a A-dominated universe {Qm = 0.3 and f^A = 0.7). This is twice larger than the 

^ . correlation length of the dark matter at z ~ 4 predicted from an analytic model by Peacock & Dodds 

QQ ' but about twice smaller than that of bright galaxies predicted by a semi-analytic model of Baugh et al. 

I We find an excess of uj{9) on small scales {9 < 5") departing from the power law fit over 3 a significance 

■ levels. Interpreting this as due to galaxy mergers, we estimate the fraction of galaxies undergoing mergers 
OO ' in our LBG sample to be 3.0 ± 0.9%, which is significantly smaller than those of galaxies at intermediate 

, redshifts. 

Subject headings: cosmology: observations — cosmology: early universe — cosmology: large-scale 
structure of universe — galaxies: high-redshift — galaxies: evolution 
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1. INTRODUCTION Field North (HDFN) over the redshift range < z < 4, 



O' Oij- rii- i- ri - ii-i J though errors in their measurements are large due to small 

• ■ Studies of clustermg properties of galaxies at high red- stat'st'cs 

^_ shifts are essential for understanding galaxy formation and ^ i- • , i i i. i 

cZj ■ . . 1 • . , , r rp/ ; rc ■ , 1 he aim of this paper IS to extend the studies on galaxy 

rrt , its relationship to structure formation. Ihe most etncient i,-; ^tt-.^ • ^ iri. 

. . , 1 i I,- u J u-ci. ^ ^ o 1 • • T clustering to z ~ 4 LBGs using a large sample of about 

• • ■ way to select high redshift (z > 2.5) galaxies is the Ly- „° , ^ . , i /n r, 
> ,. , 1 . 1 • u- u J 1 • • 1 J- J 1200 LBG candidates identified by the two color (B - i? 

man break technique. High redshift galaxies are isolated „ . ,. , , . ■ ■ ,i -i , 

■ . , 1 , • ii, • rnr 4-- j-- vs R — I ) Selection technique, ihis is the largest sam- 

. m a two-color plane using their U V continuum properties, , „ ' . ^ ■ ?• i. • ?; i . 

rN , 11- 1 i J ■ Zu- 11 J T 1, 1 pie of z ~ 4 LBGs m a contiguous area obtained to date, 

and galaxies selected m this way are called Lyman break ,;i i, i -i t j t ri;- .■ 

a ; galaxies (LBGs). Giavalisco et al. (1998) studied cluster- ^-^fbles detailed studies of clustermg properties 

■ r ^ ,. fT-nri ii u ■ f 1 1 r of galaxics at the highest redshift. throughout this paper, 

mg properties of LBGs on the basis of a large sample for •, , • , , a t-. , f f ^ 

T, ■ ^ oc c:\ T T>ri 1 • J o 1 i J u magnitudes are m the AB system, 

bright [R < 25.5) LBGs lying around z = 3 selected by ° 

UnGR colors. They found that the spatial distribution of 2. observations and data reduction 
z = 3 LBGs is strongly biased relative to the expectations 

for the dark matter in Cold Dark Matter models with a , D^^P, wide-field B-,V-,R- and i -band imaging 

linear bias of 4.5 predicted for an Einstein-de Sitter cos- ^^^ta of a central 30 x 24 area m the Subaru/XMM- 

mology (see also Adelberger et al. 1998). In a subsequent Newton Deep Survey Field (2M8"00^-5°12'00"[J2000]) 

paper, Giavahsco & Dickinson (2001; GDOl) found the ^^a^^"^ ^^^^ ^^^aru Prime Focus Camera (Suprime- 

clustering amphtude of z ^ 3 LBGs to depend on their ^am; Miyazaki et al. 1998) during the commissiomng 

rest-frame uUraviolet luminosity, with fainter galaxies be- observing runs on November 24-27 2000. The present 

ing less strongly clustered. Arnouts et al. (1999) measured ^^^^ is based on the B R, and i data. The mdividua 

the correlation length of faint galaxies in the Hubble Deep ^CD data were reduced and combined using IRAF and 

the mosaic-CCD data reduction software developed by us 

^ Based on data collected at Subaru Telescope, which is operated by the National Astronomical Observatory of Japan. 
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(Yagi 1998). The combined images for individual bands 
were aligned and smoothed with Gaussian kernels to match 
their seeing sizes. The final images cover a contiguous 618 
arcmin^ area with a PSF FWHM of 0."98. The net expo- 
sure times of the final images are 177, 58, and 45 minutes 
for B, R, and i' , respectively. The limiting magnitudes 
are B = 27.6, R = 26.5, and i' = 26.2 for a 3a detection 
on a 2" diameter aperture. Source detection and photom- 
etry are performed using SExtractor version 2.1.6 (Bertin 
& Arnouts 1996). The i'-band frame is chosen to detect 
objects, and we limit the object catalog to i' < 26, in order 
to provide a reasonable level of photometric completeness. 

3. SELECTION OF Z ^ 4 LYMAN BREAK GALAXIES 

Our catalog contains 42,557 objects with i' < 26.0 in to- 
tal. On the basis of expectations from GISSEL96 (Bruzual 
& Chariot 1993) population synthesis models ^ with dust 
attenuation of E{B — V) = — 0.5 assuming Calzetti 's 
(1997) extinction law, we define the photometric selection 
criteria for galax;ies at 2; ~ 4 as 
B-R> 2.1, R-i' < 0.5, B-R> 5.4(ii - i') + 0.9 (1) 

We estimate from the models that the expected redshift 
distribution of galaxies satisfying eq. (1) is 2 = 3.8 ± 0.5 
(Ouchi et al. 2001). 

There are a total of 1,192 objects which meet the crite- 
ria. Figure 1 shows the number counts of the objects, with- 
out correction for incompleteness of detections, against i' 
(filled circles). It is found that our counts are consistent 
with those given in Steidel et al. (1999), who selected z A 
LBGs using GRI color, and that the faint end of our counts 
matches the counts derived from the HDFN. For the fol- 
lowing analyses, we define three LBG samples; (1) the 
whole sample containing 1,192 objects, (2) the sample for 
bright {i' < 25.5) objects, and (3) the sample for faint 
(25.5 <i' < 26) objects. 

We have estimated the contamination and completeness 
of our LBG sample by Monte Carlo simulations, generat- 
ing artificial galaxies using the HDFN objects for which 
magnitudes, colors, and photometric redshifts are given in 
Fernandez-Soto, Lanzetta, & Yahil (1999), and distribut- 
ing them on our original images (Ouchi et al. 2001). The 
contamination is defined, for the detected simulated ob- 
jects, as the ratio of low redshift {z < 3.3) objects meeting 
eq.(l), to all objects satisfying eq.(l). The completeness 
is defined as the ratio of i' < 26 simulated objects passing 
our detection threshold and satisfying cq.(l), to all (de- 
tected -I- undetected) i' < 26 simulated objects. We find 
from the simulations that the completeness is 45%, 51%, 
and 40% for the whole, bright, and faint samples, respec- 
tively. Similarly, the contamination is 17%, 19%, and 14%, 
respectively. Note that the completeness defined here is a 
combination of the detection completeness and the com- 
pleteness of color selection. Our color criteria effectively 
sample HDFN galaxies at 3.3 < 2: < 4.3 with a selection 
completeness of 89%. Namely, 89% of z ~ 4 galaxies pass 
eq.(l) once they are detected. We adopt for the redshift 
distribution N{z) of LBGs in our sample the one derived 

^ The iiiodcl parameters arc chosen to match the observed colors of z ■ 
an iiistantaiiious burst (Sawicki & Yee 1998). 

The formal error does not include sample variance, which is caused from field-to-field variations. 

Assuming that the clustering pattern is fixed in comoving coordinates in the redshift range of our sample, we take the functional form, 
F{z) = (1 -h z)/{l + 3.8)"(^+^\ where e = -1.2. The eflFect of the change in e over < e < —3 on ro is, however, very small. 



from the simulations taking account of the contamination 
and completeness. 



4. ANGULAR CORRELATION FUNCTION 

Figure 2 shows the sky distribution of z ~ 4 LBGs 
in our sample. We find in this figure somewhat an in- 
homogeneous distribution of LBGs, especially for bright 
{i' < 24.5) ones. This is not due to the inhomogeneity of 
detection completeness over the image, because the Monte 
Carlo simulations show that differences in detection com- 
pleteness among small (1.'7 x 1.'7) areas in the image are 
less than 10%, and that the detection completeness does 
not correlate with the distribution of LBGs. 

We derive the angular two-point correlation function, 
iu'iO), using the estimator defined by Landy & Szalay 
(1993), ojobsiO) = [DD{9) - 2DR{e) + RR{9)]/RR{9), 
where DD{6), DR{6), and RR{9) are numbers of galaxy- 
galaxy, galaxy-random, and random-random pairs normal- 
ized by the total number of pairs in each of the three sam- 
ples. The random sample is composed of 200,000 sources 
with \\iQ same? gciometrical constraints of the data sam- 
ple. T\w formal error in w{6) is assigned by = 
■\/[l + ujobs{9)]/ RR- The real correlation function oj{9) 
is offset by the integral constant (JC: Groth & Peebles 
1977); uj{9) = ujobsiO) + IC ,where IC of the whole sample 
is calculated to be 0.0045. 

The resulting angular correlation function for the whole 
sample is shown in Figure 3 after the application of IC. 
We fit a power law, lv{9) = Ai^9~^, to the data points, and 
find the slope (3 = 0.6^04, which is consistent with those 
for nearby galaxies, /? = 0.7 — 0.8, though the errors in our 
estimate are large. The best fit values of A^^ for (3 = 0.8 
are summarized in Table 1. 

The two-point angular correlation function is related 
to the spatial correlation function ^(r) = {r/ro)^~' by 
an integral equation, the Limber transformation (Peebles 
1980;Efstathiou et al. 1991). 



F{z)Dl-''{z)N{zfg{z)dz 



N{z)dz 



(2) 



where F{z) describes the redshift dependence of ^(r 
Dg{z)\s the angular diameter distance, g{z) = -^[(1-1- 
2)2(1 -I- n^z + f2A((l + 2)-2 - l))^/2]^ and C is a numeri- 
cal constant C = x/Tr^-^^rrnU^ ■ The slope (3 is related by 



" r(7/2) 

/? = 7 — 1. The Limber transformation requires the red- 
shift distribution N{z) of LBGs (see §3). We adopt the one 
derived from the simulations based on the HDFN galaxies 
(see §3). The obtained values of ro are presented in Table 
1 for three cosmologies, = (0.3, 0.7), (0.3, 0), and 

(1,0). Those correlation lengths correspond to projected 
angular scales of 100" — 110" at z = 3.8. These scales 
fall just outside of the range where the power-law fitting 
is excellent (10" < 9 < 100"), implying that ro is reliably 
measured from the fit. Varying N{z) in a reasonable range 

: 3 galaxies: Salpeter IMF, Z^f,(„^i = 0.2Zq, and an age of 10 Myr for 
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changes the results only slightly; for example, a tophat dis- 
tribution over 3.3 < 2 < 4.3 returns 8% larger values, and 
adopting a narrower distribution, 3.4 < z < 4.2, decreases 
ro by 11%. The random contaminant objects dilute the 
amplitudes of angular correlation function, Ai^, by a fac- 
tor up to (1 — /)^, where / is the contamination (see §3). 
The contamination correction increases ro by about 20% 
at most. The effect of field-to- field variations in our sam- 
ple is probably modest, since our sample probes a large 
comoving volume, 36 x 36 x 520 = 6.7 x IQ^'Jir'-^ Mpc"^ 
for = 0.3 and = 0.7 cosmology. Observations of 
other fields and analyses of N-body simulations for struc- 
ture formation will be useful for evaluation of the effect of 
field variation. 

5. DISCUSSION & CONCLUSIONS 

Figure 4 shows the observed correlation length ro (in 
comoving units) of galaxies at (z) = 3.8 in our sample, 
together with those at various redshifts between z = 
and 3. Two data points at z = 3 are the measurements 
by GDOl for bright {R < 25.5) LBGs and faint {R < 27) 
LBGs. It is found that the ro for our whole sample is 
only slightly smaller than that for GDOl's bright LBGs at 
z = 3 ( The difference in the mean brightness in our three 
samples is less than 1 mag, and the absolute magnitudes 
probed by our samples correspond roughly to those probed 
by GDOl's bright sample). This suggests that the cluster- 
ing amplitude does not change significantly from z = 3 
to 3.8, with a possible decrease within ~ 50%. GDOl 
found strong luminosity dependence of ro for z = 3 LBGs 
as shown in Figure 4. Our sample does not contain as 
faint LBGs as they studied, but we also find possible de- 
pendence on luminosity in our sample, with fainter LBGs 
having a smaller ro. Arnouts et al. (1999) obtained a large 
value ro = (4.28 ± 1.69)h~^ Mpc for z ~ 4 galaxies in the 
HDFN. This is the opposite sense since the galaxies in the 
HDFN are fainter than those in our sample. However, the 
number of galaxies they used is only 35. 

The ro values of LBGs at z = 3 — 4 are about half that 
of the nearby galaxies. The dotted and solid lines in Fig- 
ure 4 present, respectively, the evolution of ro for the dark 
matter predicted by an analytic model and 'galaxies' of 
R < 25.5 predicted by a semi- analytic model for galaxy 
formation in a cold dark matter universe with f2m = 0.3, 
Qa = 0.7, and ag = 0.94 (Baugh et al. 1999). For the 
reader's guide, we also show by the dashed line the ro 
of the dark matter predicted by linear theory normalized 
to ro(0) = 5.3/1-1 ]y[p(, rpjjg Qf ^jjg observed LBGs 
at z = 3.8 is found to be 2.9 times larger than that for 
the dark matter given by Baugh et al. (1999) (In prac- 
tice, Baugh et al. adopted for the dark matter corre- 



lation function the analytic model of Peacock & Dodds 
1996.) Note that if one compares predicted correlation 
lengths for galaxies with observations, predictions should 
be made based on selection criteria of galaxies which are 
similar to those adopted in the observations. If we in- 
terpret this as biasing of galaxy distribution, which is 
expressed as ^galir) = b'^ ^matter {r), we obtain a linear 
bias b ~ 2.6, which is similar to that found by GDOl for 
bright z = 3 LBGs. On the other hand, the semi-analytic 
model prcdic;ts much stronger clustering for galaxies (Sim- 
ilar predictions are made by, eg, Kauffmann et al. 1999 and 
Yoshikawa et al. 2001). The predicted value at ^! = 3.8 
is twice as large as observed in this study. Further discus- 
sion requires a detailed comparison of galaxy properties, 
including selection criteria, between their simulated galax- 
ies and the LBGs in our sample. 

In Figure 3, we find an excess of uj{6) at small scales 
(9 < 5"), over 3 a significance levels, relative to the best fit 
power law. This may be caused from various effects, for ex- 
ample, galaxy-galaxy mergers, two galaxies within a com- 
mon dark matter halo, and/or field variance. If we inter- 
pret this as galaxy-galaxy mergers of those i' < 26 LBGs, 
which are classified as bright LBGs with L ° > i* 

^ 1700^ ~ 1700A 

(Ouchi et al. 2001), we calculate, following Roche & Eales 
(1999), the fraction of galaxies undergoing mergers as 
fpair = 3.0 ± 0.9% from the observed and expected num- 
bers of galaxy pairs with a separation of l."5 < 6 < 4" 
(projected physical separation corresponds to 7.5 to 20 
h~^]ipc at z = 3.8), where the expected number is calcu- 
lated from an integration over l."5 < 6* < 4" of the best fit 
power law derived for the data at ^? > 4". This fpair value 
is significantly smaller than those for intermediate-redshift 
(z ~ 0.3) galaxies whose fpair at < 20 h~^kpc is 5 — 15% 
(Carlberg, Pritchet, & Infante 1994; Infante, de Mcllo, & 
Menanteau 1996; Roche & Eales 1999). This may im- 
ply that merger rates for massive galaxies are lower at 
z ~ 4 than at present, if we assume that bright LBGs are 
massive. However, z'-band magnitudes (rest-frame UV for 
z ^ 4LBGs) arc sensitive to on-going star-formation ac- 
tivities, and those luminosities refiect intensity of current 
star-formation. Observing those LBGs in near-IR bands 
and estimating their mass are needed to address this issue. 
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misioning the Suprime-Cam that makes these difficult ob- 
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Fig. 1. — Observed surface densities of z ~ 4 LBGs in i' (or /) magnitude without correction for incompleteness of detections. Filled circles 
show our 1192 objects. Open squares are similarly color-selected objects using GRI by Steidel et al. (1999). Triangles indicate objects in the 
HDFN selected by us using eq.(l), whose magnitudes have been transformed to our BRi' magnitudes following Ouchi et al. (2001). 
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Fig. 2. — Sky distribution of our z ~ 4 LBG candidates. Different symbols correspond to different magnitude bins defined in the panel. 
Masked regions to avoid effects of bright stars are shown by dashed circles. 
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Fig. 3. — Angular correlation function for the whole sample. The solid line shows the best fit power law with u}{9) = AuiO " * 
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Fig. 4. — Correlation lengths at various redshifts in comoving units. The filled square indicates the value for our whole sample, while filled 
and open circles are those of z ~ 3 LBGs with R < 25.5 and R < 27 (GDOl). The open inverse triangle shows the upper limit value for our 
whole sample, when the sample contamination maxmumly dilutes the angular correlation function. Filled pentagon, triangles, and diamonds 
represent measurements for nearby and intermediate-redshift bright galaxies (Loveday et al. 1995, Carlberg et al. 2000, Brunner, Szalay, & 
Connolly 2000). The dashed line shows the prediction of linear theory normalized to ro(0) = 5.3h~^ Mpc. Dotted and solid lines are ro of 
dark matter and galaxies with R <25.5 (including dust extinction) predicted by a semi-analytic model (Baugh et al. 1999). 
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Table 1 

Sample definitions and correlation properties. 



Sample 


Magnitude Range 


No6/ 




{R - i'Y 




A « 


rof(Open,EdS)[/i-iMpc] 


whole 
bright 
faint 


i' < 26 
i' < 25.5 
25.5 < i' < 26 


1192 
549 
643 


25.5 
25.2 
25.8 


0.12 
0.12 
0.10 


0.165 
0.189 
0.144 


0.71 ±0.26 
0.97 ±0.57 
0.56 ±0.25 


r, 7+0.5/r, /I+0.5 -1 p,+O.Z\ 

^- ' -0.6l^-*-0.5' ^-0-0.4^ 

o o+i Ofo 9+°-^ 1 c,+°-^'\ 

2 4+°-^('2 1+°-^ 1 4+°-^~l 
^•*-0.7V^--'--0.6> ^-^-OA) 



^Number of LBG candidates. 
'^Median i' magnitude of the sample. 
'^Median R — i' color of the sample. 

'^Fraction of contaminant sources. 

"Best fit amplitude of power law function for the angular correlation function. The slope of power law is 
/3 = 7 — 1 = 0.8 (fixed). The reduced of the fitting is 2.1, 2.3, and 0.6 for the whole, bright and faint 
samples, respectively. 

^Correlation length (comoving) in a A-dominated universe (open universe and Einstein de-Sitter universe). 



